Using Visual Gestures in Combination with Commoditized Optic Sensor Technology to Replace or Augment Mouse Interfaces

This paper will explore the feasibility of applying visual gestures in conjunction with commoditized optic sensor technology to mouse interfaces. To that end, the Microsoft Kinect was used. This can be considered as an implementation detail however for the most part. A Kinect-controllable desktop simulator capable of comprehending visual gestures was developed for usability testing. The goal of the simulator is to provide users with a set of use cases they are likely to encounter when interfacing with desktop GUIs. The simulator and its visual gesture mechanisms were put to test in a field experiment involving 11 participants. From these experiments, metrics such as perceived level of user-friendliness were distilled and compared to the mouse alternative. From a functional point of view, the results of these experiments were generally in favor of the mouse. In some cases however, participants have placed visual gestures on an equal footing as mouse interfacing. In such cases, participants found visual gestures to be different, ‘cool’ and ‘fun to work with’. Even though the focal point of this paper remains largely at the usability aspect of visual gestures, a significant amount of effort has gone into getting the Kinect to work on a PC as well. Consequently, this paper will also touch on some of the caveats one is likely to encounter when trying to get the Kinect to work on a PC.